Development of an optical motion capture setup for feature extraction and statistical analysis of the pianist’s expert gestures

Movements of the body and especially of the hand are extremely complex, and are driven by several processes of biomechanical and neuroplastic types, among others. Their study presents a great interest in Human-Computer Interaction (HCI), in sportive and musical education, as well as in medicine and reeducation. The aim of this research was to capture and analyze highly complex gestures performed by experts in hand motor control, such as musicians. Musical gestures are highly formalized by indications generally noted on a musical score, allowing a more focused and thus objective study of the hands movements. Piano playing, which requires rather precise and coordinated movements, was therefore chosen as our case study. In this research, we developed an optical motion capture setup that allowed the capture of pianists' expert gestures in playing complex piano pieces. The setup consisted of 12 infrared cameras aimed at the keyboard and capturing in 3D all Degrees Of Freedom (DOF) of both hands, by means of 26 retro-re ective markers placed on each hand. With this setup, we captured performances by four pianists with di erent expertise levels, playing six di erent piano pieces and an exercise. An example of the successful capture of the Fantasy in C Major by Robert Schumann can be found here: http://youtu.be/lNwWHlkjNJ4. The exercise allowed us to analyze redundant DOF solicitations in a basic keystroke gesture according to di erent tempi and movement instructions. This analysis showed that pianists have their own movement signature in natural playing. It also showed that the same pianist can use its redundant DOF of the arm and hand di erently, and thus play the same musical pattern in countless di erent ways. We also carried out a Principal Component Analysis (PCA) of the hands gestures, based on the six pieces performed by the four pianists. Results of PCA showed the power of this tool for dimensionality reduction, and representation of pianists' expert gestures. Eight Principal Components (PC) were generally su cient to represent 95% of the 23 DOF of the hand (and only three PC for 80%). Furthermore, PCA allowed us to decompose hands movements into eigengestures associated with each PC (see http://youtu.be/6qIHn9UUJW4). Finally, both analyses of the exercise and of the piano pieces seemed to show that a more trained pianist has a ner hand motor control, and in particular, a ner balance between the two hands. Results of this study show that PCA is a powerful tool for motion analysis and could be applied in di erent elds such as HCI, and sportive and musical education. Furthermore, in a clinical context, PCA could provide an assessment tool for patients su ering from motor disability.